Method of coordinating a path switch and network elements associated therewith

ABSTRACT

In one embodiment, the method includes instructing, by a controller, a switch to change from sending data via a first tunnel to sending data via a second tunnel. The first tunnel is between the switch and a first network element, and the second tunnel is between the switch and a second network element. The method further includes receiving, by the controller, acknowledgement from the switch, and notifying, by the controller, the second network element that packets will no longer be sent via the first tunnel in response to the received acknowledgement.

BACKGROUND

FIG. 1 illustrates an example of a portion of a prior art 3GPP wirelessnetwork. In 3GPP wireless networks, path switch coordination isaccomplished via the sending of an “end-marker” that indicates nofurther packets will be sent on a path. An end-marker is sent in orderto assist the packet reordering function in the target base station(e.g., enhanced NodeB also referred to as eNodeB or eNB). The servinggateway (SGW) for inter-eNB handoffs sends the end-marker whenever theserving eNB changes for a radio access bearer (RAB). In the 3GPPimplementation, the “end-marker” is a message signaled in the GPRStunneling protocol (GTP) header and the packet does not contain userdata. When the SGW changes, the packet gateway (PGW), which communicatesuser data from a content source, sends an end-marker message to the SGWvia the GTP-U S5/S8 interface. Upon reception of an end-marker message,the eNB or SGW may release the bearer plane resources for that leg.

SUMMARY

While this solution is fine for 4G path switching, it is not compatiblewith evolving software defined networking (SDN) architectures that areproposed for 5G. In those architectures signaling functions such as the“end-marker” belong in the control plane, supported by a centralized SDNcontroller. Namely, the switches (corresponding in part to the SGW)cannot send end-markers in 5G architectures.

At least one example embodiment relates to methods and/or apparatusesfor providing end-marker functionality in a software defined networkingarchitecture. While example embodiments are applicable to the evolving5G wireless standard, example embodiments may also be applied innon-wireless scenarios where packets are forwarded via multiple pathsand a reordering function needs to know when the last packet has beenreceived on a given path.

In one embodiment, the method includes instructing, by a controller, aswitch to change from sending data via a first tunnel to sending datavia a second tunnel. The first tunnel is between the switch and a firstnetwork element, and the second tunnel is between the switch and asecond network element. The method further includes receiving, by thecontroller, acknowledgement from the switch, and notifying, by thecontroller, the second network element that packets will no longer besent via the first tunnel in response to the received acknowledgement.

In one embodiment, the notifying includes sending a message to thesecond network element, the message indicating that no further packetswill be sent via the first tunnel.

In one embodiment, the method further includes delaying the sending.

In one embodiment, the receiving receives a sequence number from theswitch, and the sequence number being for a last packet that the switchsent via the first tunnel.

In one embodiment, the notifying includes sending the sequence number.

In one embodiment, the sequence number is part of the acknowledgement.

In one embodiment, the method further includes notifying, by thecontroller, the first network element that packets will no longer besent via the first tunnel in response to the received acknowledgement.

In one embodiment, the instructing instructs the switch to change fromsending data via the first tunnel to sending data via the second tunnelin response to a serving base station for a mobile station changing fromthe first network element to the second network element

In one embodiment of a controller, the controller includes a memorystoring a program routine, at least one processor configured byexecuting the program routine to, instruct a switch to change fromsending data via a first tunnel to sending data via a second tunnel, thefirst tunnel being between the switch and a first network element, andthe second tunnel being between the switch and a second network element;receive acknowledgement from the switch; and notify the second networkelement that packets will no longer be sent via the first tunnel inresponse to the received acknowledgement.

In one embodiment, the controller is configured to perform the notifyingby sending a message to the second network element, the messageindicating that no further packets will be sent via the first tunnel.

In one embodiment, the controller is configured to delay sending themessage.

In one embodiment, the controller is configured to receive a sequencenumber from the switch, the sequence number being for a last packet thatthe switch sent via the first tunnel.

In one embodiment, the controller is configured to perform the notifyingby sending the sequence number.

In one embodiment, the sequence number is part of the acknowledgement.

In one embodiment, the controller is configured to notify the firstnetwork element that packets will no longer be sent via the first tunnelin response to the received acknowledgement.

In one embodiment, the controller is configured to instruct the switchto change from sending data via the first tunnel to sending data via thesecond tunnel in response to a serving base station for a mobile stationchanging from the first network element to the second network element.

Another embodiment of the method includes receiving, by a switch, aninstruction to change from sending data via a first tunnel to sendingdata via a second tunnel, the first tunnel being between the switch anda first network element, and the second tunnel being between the switchand a second network element; tearing down the first tunnel by theswitch; and sending, by the switch, an acknowledgement signifying thatno further data packets will be sent via the first tunnel after thefirst tunnel is torn down. The first and second network elements may befirst and second base stations, respectively.

In one embodiment, the sending includes sending a sequence number of alast data packet sent via the first tunnel.

In another embodiment, the method includes receiving first packets froma first network element at a second network element; receiving, at thesecond network element, second packets from a switch; receiving, at thesecond network element, a notification from a controller that no morepackets will be received from the first network element; andde-allocating, at the second base station, resources for receiving thefirst packets in response to the notification. The first and secondnetwork elements may be first and second base stations, respectively.

In one embodiment, this method further includes managing, at the secondnetwork element, the first and second packets by eliminating duplicatepackets and ordering the first and second packets

BRIEF SUMMARY OF THE DRAWINGS

The present invention will become more fully understood from thedetailed description given herein below and the accompanying drawings,wherein like elements are represented by like reference numerals, whichare given by way of illustration only and thus are not limiting of thepresent invention and wherein:

FIG. 1 illustrates an example of a portion of a prior art 3GPP wirelessnetwork.

FIG. 2 illustrates an example of a portion of a software definednetworking wireless network architecture.

FIG. 3 illustrates the components of a network element according to anexample embodiment.

FIG. 4 illustrates a communication flow diagram for the method ofcoordinating a path switch according to an example embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Various example embodiments will now be described more fully withreference to the accompanying drawings in which some example embodimentsof the invention are shown.

Detailed illustrative embodiments are disclosed herein. However,specific structural and functional details disclosed herein are merelyrepresentative for purposes of describing example embodiments of thepresent invention. This invention however, may be embodied in manyalternate forms and should not be construed as limited to only theembodiments set forth herein.

It will be understood that, although the terms first, second, etc. maybe used herein to describe various elements, these elements should notbe limited by these terms. These terms are only used to distinguish oneelement from another. For example, a first element could be termed asecond element, and, similarly, a second element could be termed a firstelement, without departing from the scope of example embodiments of thepresent invention. As used herein, the term “and/or,” includes any andall combinations of one or more of the associated listed items.

It will be understood that when an element is referred to as being“connected,” or “coupled,” to another element, it can be directlyconnected or coupled to the other element or intervening elements may bepresent. In contrast, when an element is referred to as being “directlyconnected,” or “directly coupled,” to another element, there are nointervening elements present. Other words used to describe therelationship between elements should be interpreted in a like fashion(e.g., “between,” versus “directly between,” “adjacent,” versus“directly adjacent,” etc.).

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of exampleembodiments of the invention. As used herein, the singular forms “a,”“an,” and “the,” are intended to include the plural forms as well,unless the context clearly indicates otherwise. It will be furtherunderstood that the terms “comprises,” “comprising,” “includes,” and/or“including,” when used herein, specify the presence of stated features,integers, steps, operations, elements, and/or components, but do notpreclude the presence or addition of one or more other features,integers, steps, operations, elements, components, and/or groupsthereof.

It should also be noted that in some alternative implementations, thefunctions/acts noted may occur out of the order noted in the figures.For example, two figures shown in succession may in fact be executedsubstantially concurrently or may sometimes be executed in the reverseorder, depending upon the functionality/acts involved.

Specific details are provided in the following description to provide athorough understanding of example embodiments. However, it will beunderstood by one of ordinary skill in the art that example embodimentsmay be practiced without these specific details. For example, systemsmay be shown in block diagrams in order not to obscure the exampleembodiments in unnecessary detail. In other instances, well-knownprocesses, structures and techniques may be shown without unnecessarydetail in order to avoid obscuring example embodiments.

Also, it is noted that example embodiments may be described as a processdepicted as a flowchart, a flow diagram, a data flow diagram, astructure diagram, or a block diagram. Although a flowchart may describethe operations as a sequential process, many of the operations may beperformed in parallel, concurrently or simultaneously. In addition, theorder of the operations may be re-arranged. A process may be terminatedwhen its operations are completed, but may also have additional stepsnot included in the figure. A process may correspond to a method, afunction, a procedure, a subroutine, a subprogram, etc. When a processcorresponds to a function, its termination may correspond to a return ofthe function to the calling function or the main function.

Moreover, as disclosed herein, the term “memory” may represent one ormore devices for storing data, including random access memory (RAM),magnetic RAM, and/or other machine readable mediums for storinginformation. The term “storage medium” may represent one or more devicesfor storing data, including read only memory (ROM), random access memory(RAM), magnetic RAM, magnetic disk storage mediums, optical storagemediums, flash memory devices and/or other machine readable mediums forstoring information. The term “computer-readable medium” may include,but is not limited to, portable or fixed storage devices, opticalstorage devices, wireless channels, and various other mediums capable ofstoring, containing or carrying instruction(s) and/or data.

Furthermore, example embodiments may be implemented by hardware,software, firmware, middleware, microcode, hardware descriptionlanguages, or any combination thereof. When implemented in software,firmware, middleware or microcode, the program code or code segments toperform the necessary tasks may be stored in a machine or computerreadable medium such as a storage medium. A processor(s) may perform thenecessary tasks.

A code segment may represent a procedure, a function, a subprogram, aprogram, a routine, a subroutine, a module, a software package, a class,or any combination of instructions, data structures, or programstatements. A code segment may be coupled to another code segment or ahardware circuit by passing and/or receiving information, data,arguments, parameters, or memory contents. Information, arguments,parameters, data, etc. may be passed, forwarded, or transmitted via anysuitable means including memory sharing, message passing, token passing,network transmission, etc.

As used herein, the term “mobile station” may be considered synonymousto, and may hereafter be occasionally referred to, as a client, mobile,mobile unit, user agent, mobile user, user equipment (UE), subscriber,user, remote station, access agent, user agent, receiver, etc., and maydescribe a remote user of network resources in a communications network.Furthermore, the term “user agent” may include any type ofwireless/wired device such as consumer electronics devices, smartphones, tablet personal computers, personal digital assistants (PDAs),desktop computers, and laptop computers, for example.

The term “base station” may be understood as a one or more cell sites,base stations, base transceiver stations, access points, and/or anyterminus of radio frequency communication. Although current systemarchitectures may consider a distinction between mobile/user devices andaccess points/cell sites, the example embodiments described hereaftermay generally be applicable to architectures where that distinction isnot so clear, such as ad hoc and/or mesh system architectures, forexample.

Example embodiments are discussed herein as being implemented in asuitable computing environment. Although not required, exemplaryembodiments will be described in the general context ofcomputer-executable instructions, such as program modules or functionalprocesses, being executed by one or more computer processors or CPUs.Generally, program modules or functional processes include routines,programs, objects, components, data structures, etc. that performsparticular tasks or implement particular data types. The program modulesand functional processes discussed herein may be implemented usingexisting hardware in existing communication networks. For example,program modules and functional processes discussed herein may beimplemented using existing hardware at existing network elements orcontrol nodes. Such existing hardware may include one or more digitalsignal processors (DSPs), application-specific-integrated-circuits,field programmable gate arrays (FPGAs) computers or the like.

FIG. 2 illustrates an example of a portion of a software defined (SDN)networking wireless network architecture. As shown, a SDN controller 50programs an SDN switch 30 to forward packets destined for a mobilestation 5 to a current serving base station, first base station 10. Thefirst base station 10 wirelessly communicates the data in the datapackets to the mobile station 5.

The SDN switch 30 forwards the packets over a first general routingencapsulation (GRE) tunnel 80. The SDN switch 30 may receive the packetsfrom one or more content servers 40 via one or more intermediaryswitches 35.

As will be appreciated, the SDN controller 50 may perform variousfunctions such as those associated with network configuration, networktopology, resource utilization, etc. In particular the SDN controller 50and associated applications may perform the functions of a mobilitymanagement entity (MME). Here, the SDN controller 50 manages sessionstates, authentication, paging, mobility, roaming, bearer managementfunctions, etc.

The various MME functionalities are well-known and will not be discussedin detail. These functionalities are referenced here to provide a properunderstanding that the SDN controller 50 manages mobility of the mobilestation 5 within the network. For instance, the SDN controller 50 mayinstruct the switch 30 via the control plane to change from sending data(e.g., packets) via the data plane for the mobile station 5 via thefirst GRE tunnel 80 to sending the packets via a second GRE tunnel 90,the second GRE tunnel 90 being between the switch 30 and a second basestation 20. For example, such a second GRE tunnel 90 may be establishedfor transitioning the serving base station from the first base station10 to the second base station 20. This action may take place, forexample, if the mobile station 5 may be handed over to the second basestation 20.

The SDN controller 50 will also instruct via the control plane the firstbase station 10 and the second base station 20 that the first basestation 10 is to forward packets received for the mobile station 5 tothe second base station 20. The first base station 10 establishesresources for forwarding the packets to the second base station 20, andthe second base station 20 establishes resources for receiving theforwarded packets. The second base station 20 also manages the packetsfor the mobile station 5, which were received from the switch 30 and thefirst base station 10, by eliminating duplicate packets and ordering thereceived packets.

In response to the instruction, and after establishing the second GREtunnel 90 if the second GRE tunnel 90 has not already been established,the switch 30 will tear down the first GRE tunnel 80. Then, the switch30 sends an acknowledgement via the control plane to the SDN controller50 signifying that no further data packets will be sent via the firstGRE tunnel 80.

When the acknowledgment is received from the switch 30, the SDNcontroller 50 sends a notification via the control plane to the firstbase station 10 and the second base station 20 that packets will nolonger be sent via the first GRE tunnel 80. In response, the first basestation 10 de-allocates forwarding resources for forwarding packet forthe mobile station 5 to the second base station 20, and the second basestation 20 de-allocates resources for receiving the packets from thefirst base station 10.

The acknowledgement from the switch 30 and the notification message fromthe SDN controller 50 may optionally include a sequence numbercorresponding to the last packet sent by the switch 30 on the first GREtunnel 80. The SDN controller 50 may include this sequence number in thenotification message to the second base station 20, and optionally, tothe first base station 10. The second base station 20 may stop packetreordering and de-allocate resources associated with the first GREtunnel 80 after a packet with that sequence number has been received.When a sequence number of the last packet will not be sent in thenotification message, the SDN controller 50 may delay sending thenotification message for a period of time after receipt of theacknowledgement from the switch 30. This helps ensure that the lastin-flight packets have been received at the second base station 20 onthe forwarding tunnel from the first base station 10.

The switches 30 and 35 may be any well-known software defined switch.The switches 30 and 35 may include one or more processors, variousinterfaces, a computer readable medium, and (optionally) a displaydevice. The one or more interfaces may be configured to transmit/receive(wireline or wirelessly) data signals via the data plane to/from one ormore other switches, servers 40 and/or base stations 10 and 20; and totransmit/receive (wireline or wirelessly) controls signals via thecontrol plane to/from the controller 50.

Server 40 may include one or more servers (i.e., a physical computerhardware system) that is configured to provide services for usersconnected to a network. Furthermore the server may constitute acorresponding node such as another mobile station 5. The one or moreservers 40 may employ connection-oriented protocols such as SessionInitiation Protocol (SIP), HTTP, and TCP/IP, and includes servers thatuse connectionless protocols such as User Datagram Protocol (UDP) andInternet Packet Exchange (IPX).

The base stations 10 and 20 may include one or more processors, variousinterfaces including one or more transmitters/receivers connected to oneor more antennas, a computer readable medium, and (optionally) a displaydevice. The one or more interfaces may be configured to transmit/receive(wireline and/or wirelessly) data or controls signals via respectivedata and control planes to/from one or more switches, SDN controllers,other base stations, and mobile stations.

The SDN controller may execute on one or more processors, variousinterfaces including one or more transmitters/receivers connected to oneor more antennas, a computer readable medium, and (optionally) a displaydevice. The one or more interfaces may be configured to transmit/receive(wireline and/or wirelessly) control signals via the control planeto/from one or more switches and the base stations.

Collectively, the base stations, the switches and the SDN controller maybe referred to as network elements. FIG. 3 illustrates the components ofa network element according to an example embodiment. For the purposesof description, the network element in FIG. 3 will be assumed to be theSDN controller 50.

As shown, the SDN controller 50 includes a processor 110, connected to amemory 120 and various interfaces 130. During operation, memory 120includes an operating system and any other routines/applications forproviding the functionalities of the SDN controller (e.g., MMEapplications). In some embodiments, SDN controller 50 may include manymore components than those shown in FIG. 3. However, it is not necessarythat all of these generally conventional components be shown in order todisclose the illustrative embodiment.

Memory 120 may be a computer readable storage medium that generallyincludes a random access memory (RAM), read only memory (ROM), and/or apermanent mass storage device, such as a disk drive. Memory 120 alsostores operating system and any other routines/modules/applications forproviding the functionalities of the SDN controller (e.g., MMEapplications). These software components may also be loaded from aseparate computer readable storage medium into memory 120 using a drivemechanism (not shown). Such separate computer readable storage mediummay include a disc, tape, DVD/CD-ROM drive, memory card, or other likecomputer readable storage medium (not shown). In some embodiments,software components may be loaded into memory 120 via one of the variousinterfaces 130, rather than via a computer readable storage medium.

Processor 110 may be configured to carry out instructions of a computerprogram by performing the basic arithmetical, logical, and input/outputoperations of the system. Instructions may be provided to processor 110by memory 120.

The various interfaces 130 may include computer hardware components thatconnect SDN controller 50 via a wired or wireless connection to theswitches 30 and 35, base stations 10 and 20, etc.

As will be understood, the interfaces 130 and programs stored in thememory 120 to set forth the special purpose functionalities of thenetwork element will vary depending on the network element.

Next, the method of coordinating a path switch according to an exampleembodiment will be described with respect to the communication flowdiagram illustrated in FIG. 4. For the purposes of description, themethod will be described as implemented on the architecture of FIG. 2.Accordingly, it will be understood that the processors in the basestations, switches and controller are configured based onroutines/modules/applications stored in associated memories to performthe special purpose functionalities associated with the method ofcoordinating a path switch.

As shown in FIG. 4, the SDN controller 50 sends an instruction to theswitch 30 to change from sending data packets destined for the mobilestation 5 via the first GRE tunnel 80 to sending data packets via thesecond GRE tunnel 90. In this example, it is assumed that the SDNcontroller 50 had previously instructed the switch 30 to establish thesecond GRE tunnel 90 as well as instructed the first base station 10 toforward data packets for the mobile station 5 to the second base station20. However, it will be understood, that the switch 30 may insteadestablish the second GRE tunnel 90 in response to the instruction, andthat the SDN controller 50 may concurrently establish packet forwardingfrom the first base station 10 to the second base station 30.

In response to the instruction, the switch 30 tears down the first GREtunnel 80, and then sends an acknowledgement (ACK) to the SDN controller30. In response to the acknowledgement, the controller 50 notifies thefirst and second base stations 10 and 20 that packets for the mobilestation 5 will no longer be sent via the first GRE tunnel 80. Thesenotifications may be sent concurrently, the notification for the firstbase station 10 may be sent first, or the notification for the secondbase station 20 may be sent first.

In response to the notification received at the first base station 10,the first base station 10 de-allocates forwarding resources forforwarding packet for the mobile station 5 to the second base station20; and in response to the notification received at the second basestation 20, the second base station 20 de-allocates resources forreceiving the packets from the first base station 10. Alternatively, ifthe last sequence number is provided with the notification, thende-allocation occurs in response to receiving the packet with the lastsequence number.

Accordingly, example embodiments provide methods and/or apparatuses forproviding end-marker functionality in a software defined networkingarchitecture. While example embodiments are applicable to the evolving5G wireless standard, example embodiments may also be applied innon-wireless scenarios where packets are forwarded via multiple pathsand a reordering function needs to know when the last packet has beenreceived on a given path.

The invention being thus described, it will be obvious that the same maybe varied in many ways. Such variations are not to be regarded as adeparture from the invention, and all such modifications are intended tobe included within the scope of the present invention.

We claim:
 1. A method of coordinating a path switch, comprising:instructing, by a controller, a switch to change from sending data via afirst tunnel to sending data via a second tunnel, the first tunnel beingbetween the switch and a first network element, and the second tunnelbeing between the switch and a second network element; receiving, by thecontroller, acknowledgement from the switch; and notifying, by thecontroller, the second network element that packets will no longer besent via the first tunnel in response to the received acknowledgement.2. The method of claim 1, wherein the notifying includes sending amessage to the second network element, the message indicating that nofurther packets will be sent via the first tunnel.
 3. The method ofclaim 2, further comprising: delaying the sending.
 4. The method ofclaim 1, wherein the receiving receives a sequence number from theswitch, the sequence number being for a last packet that the switch sentvia the first tunnel.
 5. The method of claim 4, wherein the notifyingincludes sending the sequence number.
 6. The method of claim 4, whereinthe sequence number is part of the acknowledgement.
 7. The method ofclaim 1, further comprising: notifying, by the controller, the firstnetwork element that packets will no longer be sent via the first tunnelin response to the received acknowledgement.
 8. The method of claim 1,wherein the instructing instructs the switch to change from sending datavia the first tunnel to sending data via the second tunnel in responseto a serving base station for a mobile station changing from the firstnetwork element to the second network element
 9. A controller,comprising: a memory storing a program module; at least one processorconfigured by executing the program routine to, instruct a switch tochange from sending data via a first tunnel to sending data via a secondtunnel, the first tunnel being between the switch and a first networkelement, and the second tunnel being between the switch and a secondnetwork element; receive acknowledgement from the switch; and notify thesecond network element that packets will no longer be sent via the firsttunnel in response to the received acknowledgement.
 10. The controllerof claim 9, wherein the controller is configured to perform thenotifying by sending a message to the second network element, themessage indicating that no further packets will be sent via the firsttunnel.
 11. The controller of claim 10, wherein the controller isconfigured to delay sending the message.
 12. The controller of claim 9,wherein the controller is configured to receive a sequence number fromthe switch, the sequence number being for a last packet that the switchsent via the first tunnel.
 13. The controller of claim 12, wherein thecontroller is configured to perform the notifying by sending thesequence number.
 14. The controller of claim 12, wherein the sequencenumber is part of the acknowledgement.
 15. The controller of claim 9,wherein the controller is configured to notify the first network elementthat packets will no longer be sent via the first tunnel in response tothe received acknowledgement.
 16. The controller of claim 9, wherein thecontroller is configured to instruct the switch to change from sendingdata via the first tunnel to sending data via the second tunnel inresponse to a serving base station for a mobile station changing fromthe first network element to the second network element.
 17. A method ofswitching a data path, comprising: receiving, by a switch, aninstruction to change from sending data via a first tunnel to sendingdata via a second tunnel, the first tunnel being between the switch anda first network element, and the second tunnel being between the switchand a second network element; tearing down the first tunnel by theswitch; and sending, by the switch, an acknowledgement signifying thatno further data packets will be sent via the first tunnel after thefirst tunnel is torn down.
 18. The method of claim 17, wherein thesending includes sending a sequence number of a last data packet sentvia the first tunnel.
 19. The method of claim 18, wherein the first andsecond network elements are first and second base stations,respectively.
 20. A method of operating a network element, comprising:receiving first packets from a first network element at a second networkelement; receiving, at the second network element, second packets from aswitch; receiving, at the second network element, a notification from acontroller that no more packets will be received from the first networkelement; and de-allocating, at the second base station, resources forreceiving the first packets in response to the notification.
 21. Themethod of claim 20, further comprising: managing, at the second networkelement, the first and second packets by eliminating duplicate packetsand ordering the first and second packets.
 22. The method of claim 20,wherein the first and second network elements are first and second basestations, respectively.